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Abstract 


Improving the proficiency in mathematical literacy is one of the efforts to improve the human resources 
quality. Its development describes the students’ strengths and weaknesses to support the improvement. 
The research tried to describe the development of students ’ proficiency in mathematical literacy, based on 
the school level and the students’ gender. The research was a cross-sectional developmental study. The 
population was all of students ranging between 13 and 16 years old in the Province of Yogyakarta Special 
Region, Indonesia. In the sample were 1,001 students in the 8", 9" and 10" grade, determined to use the 
combination of stratified and the cluster random sampling technique. The data were collected using 30 
items. The data analysis was conducted by estimating students’ ability through the item-response theory 
approach under the Rasch model and by using the parameters that had been adjusted to the international 
study item parameter under the concordance model for the linking score to interpret the students’ 
development. The research results showed that there has been improvement of literacy proficiency among 
the 8", 9" and 10" grade students. The development would generate higher scores utilizing multiple choice 
test items. There was a tendency that the school level, gender, the form of items, content, context, and 
process literacy would influence the mathematical literacy score. The efforts for improving proficiency in 
mathematical literacy were discussed. 

Keywords: ability development, mathematical literacy, PISA released items, cross-sectional 
developmental study 


Introduction 


The efforts of improving the human resources quality, improving the literacy proficiency 
still becomes the focus of attention for all nations. Similarly, Indonesia as a developing 
country has also been pursuing the improvement of the human resources quality and that 
includes the students’ proficiency in the schools. The proficiency development, especially 
that of the students, has still been pursued in order to overcome the learning results that have 
not been satisfying based on the results of TIMSS (Trends in International Mathematics and 
Science Study) (Mullis, Martin, Foy, & Arora, 2012; Mullis, Martin, Ruddock, O’Sullivan, & 
Preuschoff, 2009), PISA (Programme for International Student Assessment) (OECD, 2014) and 
the results of national study such as the national examination (Kemendikbud Balitbang, 2014; 
2015; 2016) and to recollect the importance of mathematical literacy mastery that correlates to 
another proficiency such as the statistical ability (Lai, Tanner, & Stevens, 2011). 

Literacy proficiency has been an international issue in education. The proficiency has 
been assessed by an international study of PISA. The objectives of PISA international study 
are to assess the students’ knowledge and skills in the real world and to prepare the students 
in the long-life learning and the community participation (Shiel, Perkins, Close, & Oldham, 
2007). The PISA international study might be implemented by the government to, for example, 
monitor the educational system (Stacey, 2011). In the PISA international study, there are three 
aspects of literacy that will be measured, namely the reading literacy, the mathematical literacy 
and the scientific literacy (OECD, 2014). 
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Mathematical literacy relates to the individual ability in formulating, identifying, 
understanding and implementing the mathematical foundations in multiple contexts that an 
individual will need to deal with in the daily life problems (OECD, 2014). The mathematical 
literacy includes the mathematical reasoning, the mathematical concept, the mathematical 
procedure, and the mathematical fact and these aspects will be used for explaining and predicting 
a phenomenon by emphasizing the competencies of process, content and context. The process 
competencies in mathematical literacy are formulating, employing and interpreting (OECD, 
2014). Then, the substances of the content that will be assessed include change and relationship, 
space and shape, quantity, uncertainty, and data. Next, the context proficiency relates to the 
personal, the occupational, the societal and the scientific context. In order to measure the context 
competencies, a researcher should implement several types of test items. Shiel, Perkins, Close 
and Oldham (2007) stated that the test item designs for the PISA assessment format were the 
multiple-choice (MC) and constructed response (CR) items. After the students’ responses have 
been analyzed, the individual ability will be classified into 7 levels, starting from below Level, 
Level 1 to Level 6. 

Kariuki, Kibet, Muthaa, and Nkonke (2012) stated that the lack of students’ performance 
could be caused by teachers’ and students’ factor such as motivation, quality of teaching and 
learning, and teaching aids and material. The students’ gender also influenced the performance 
in mathematics. The males’ performance was higher than the females’ performance (Hall, 
2012; Zhu, 2007). However, these opinions are different than the results of a study by Ajali and 
Imoko (2015), who stated that there was indifferent achievement significantly between male 
and female. Furthermore, there is another factor that might influence the learning achievement 
namely the quality schooling (Bohlman & Pretorius, 2008). 

There are multiple efforts that might be pursued in order to improve the literacy 
proficiency. These efforts are pursued by the government, the institutions and even the 
researchers themselves. Starting from the strategy for developing the literacy programs, the 
Indonesian government has implemented multiple principles, benefitted the local context, the 
local design, the participatory process, the action outcome (Department of National Education 
Indonesia, 1999), and benefitted the scientific approach (Nurhariyanti, Waluyo, & Wardono, 
2015). Another effort that might be pursued is integrating the classroom literacy learning 
(Steihiliber, 2011) and suggesting the integration of mathematical literacy into other school 
subjects (Bowie & Frith, 2006). 


Problem of Research 


Within the learning process, mathematics should be mastered by the students gradually 
from one year to another. Similarly, the students develop the mathematical literacy proficiency 
gradually from one year to another. The expectation is that the students’ literacy proficiency 
will be increasing over time. The development of literacy proficiency should be noticed in 
order to provide feedbacks for the teachers and the policymakers and to evaluate and perform 
the improvement toward the learning process that has been conducted and the importance of 
policy refinement including the curriculum for improving the students’ mathematical literacy 
and educational literacy in general. Because multiple factors influenced the low students’ 
achievement in learning mathematics such as school level, test type, and students’ gender, the 
development of mathematical literacy is described based on the factors. 
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Research Focus 


The research was to describe the development of junior and senior high school students’ 
mathematical literacy proficiency by considering the variables of school level, test type, and 
students’ gender. 


Research Methodology 
Design 


The research was an explorative descriptive research with the cross-sectional type of 
developmental research design. In the research, the researchers would like to describe the 
development of 8", 9"" and 10" grade students’ mathematical literacy proficiency based on the 
data. The data was collected in 2015 and analyzed in 2017-2018. 


Population and Sample 


The population in this research was all of the junior high school students ranging between 
13 and 15 years old and the senior high school students ranging between 15 and 16 years old 
in the Province of Yogyakarta Special Region, Indonesia. The sample collecting technique was 
the combination of the stratified random sampling technique and the cluster random sampling 
technique. The sample members in the research were the 8" and 9" grade students from junior 
high schools and the 10" grade students from senior high schools; the schools consisted of high- 
performance, moderate-performance and low-performance category based on the scores in the 
Mathematics National Examination. The number of the sample members was 1,001 students 
consisting of 155 students of 8" grade who were between 13 and 14 years old, 386 students of 
9" grade students who were between 14 and 15 years old and 460 students of 10" grade students 
who were between 15 and 16 years old. 

When collecting data, the researchers told the students about the research. The result 
of the test that was finished by the students was only to measure the students’ proficiency in 
mathematical literacy. The result of the test wouldn’t influence anything in relation to students’ 
achievements and judgments. All of the students’ identities were hidden to guarantee the 
objectivity of the research and students’ safety and confidentiality. 


Instrument and Procedures 


The data were collected by administering the test. The test items in the test instrument 
were designed by adopting the PISA test items that had been released; the length of the test was 
30 items. The PISA test items from 3 periods (2003, 2007 and 2011) that had been released were 
translated into Indonesian language and the contexts were replaced to the ones that govern in 
Indonesia. The validity of the test items had been validated while the reliability of the test items 
had been estimated. 

The validity of the test instrument had been validated using the content validity that 
was conducted in order to identify the relevance and the representativeness of the instrument 
toward the domain under assessment. The validation toward the test instrument was conducted 
by consulting the text instrument to the experts (professional judgments) in relation to the 
domains of content and context and the domain of process in the PISA test-item models. The 
experts also provided feedbacks regarding the material truth, the composition of substances in 
each domain, the test item readability and the relevance between the test item context and the 
students in Indonesia. 
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In order to estimate the reliability of the test in the form of essay or multiple choices 
with the dichotomous data, the researcher implemented the Cronbach’s Alpha. The test was 
administered toward 30 students from the 7" Grade in the 1 State Junior High School Bantul, 
Yogyakarta, Indonesia. The index of reliability was equal to 0.707 and the standard error of 
measurement (SEM) was equal to 2.81. Based on the SEM score, the researchers implied that 
if the PISA test item model had been administered again then the score that the students would 
attain would be from X,,— 2.81 to X,,+ 2.81. 


Data Analysis 


For the data analysis, the researcher estimated the students’ literacy proficiency by 
implementing the item response theory under the Rasch model and by using the parameters that 
had been equalized into the international study test item with the concordance model for the 
linking score. The concordance was conducted by implementing the Mean and Mean Method 
and then the researcher interpreted the inter-year proficiency literacy development. 

The steps of data analysis were as follows: (1) estimating the item parameters and the 
ability parameters by operating the Rasch model proposed by Masters (2010) both for the 
students’ response from the multiple-choice test items and the dichotomous and polytomous 
constructed response test item with the assistance of CONQUEST program (Wu, Adam, & 
Wilson, 1997) and the calibration concurrent model for the 8", 9" and 10" grade; (2) adjusting 
the test item parameters to the international test item parameters by means of Mean and Mean 
method (Hambleton & Swaminathan, 1985) until the researcher attained the adjustment of the 
item parameter to the international scale for the 8", 9 and 10" grade and the results of the 
adjustment are displayed in Table 1; (3) implementing the adjustment equation from the second 
step in order to adjust the ability parameter for each grade; (4) performing a descriptive analysis 
in order to present the students’ proficiency development for the 8", 9" and 10" grade; and (5) 
categorizing the participants’ proficiency in accordance with the data analysis within the PISA 
model for each classroom by using the estimation results of participants’ ability. 

To categorize the participants’ proficiency based on the data analysis within the PISA 
model, the students’ proficiency scale should be transformed using a mean that was equal to 
500 and the standard deviation that was equal to 100. The results of the transformation then 
would be recoded into 7 levels that consisted of below level 1, level 1, until level 6 similar with 
what stated in the Technical Report (OECD, 2014). Utilizing the results of transformation and 
the categorization of literacy proficiency, the researcher monitored the development trend. The 
literacy development trend then was described by referring to the grade and by considering the 
variables of the school level, gender and item form. 


Table 1. The adjustment of the proficiency parameter into the PISA international 








parameter. 
International Mean of Difficulty Equation of 
: 2 Mean of Proficiency 
Domain Sub-Domain soe: Parameter Resulted Beta 
Difficulty : Scale 
in the Research ‘ 
Parameter Conversion 
General 
Mathematical -0.235 -0.446 0.211 0.021 
Literacy Proficiency 
Test Item Design Multiple Choice -1.287 -2.293 1.006 
oo 0.567 0.520 0.045 
Response 
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The development of students’ mathematical literacy within 8", 9" and 10" grade, 
overall, was presented in Figure 1. 
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Figure 1. The development of students’ mathematical literacy proficiency based 
on grade. 


Based on Figure 1, the score of the 8" grade students’ mathematical literacy is equal 
to 314.53, the score of the 9" grade students’ mathematical literacy is equal to 430.97 and 
the score of the 10 grade students’ mathematical literacy is equal to 450.62. These scores 
showed that there had been improvement in the students’ mathematical literacy proficiency as 
the students got promoted to the higher grade. Then, the researchers elaborated the students’ 
mathematical literacy proficiency according to the types of the test that had been administered. 
The test model PISA that had been administered consisted of the multiple-choice test items and 
the constructed response test items (the test items which answers were structured). Based on 
the type of the test, the description on the 8", 9" and 10" grade students’ mathematical literacy 
proficiency would be as follows. 
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Figure 2. The development of students’ mathematical literacy proficiency score 
based on the grade and test item type. 


Based on Figure 2, there are differences among the students’ mathematical literacy 
proficiency in completing the constructed response-type multiple choice (MC) test items. The 
84, 9 and 10" grade students are more inclined to complete the multiple-choice test items than 
the constructed response (CR) ones. It was found that the students got high scores after they 
completed the multiple choice test items; the score was higher than the international median 
which equals to 500. The situation was very contrast and significantly different from the time 
when they completed the constructed response test items in which they scored below 200. 
However, the 8", 9" and 10" grade students had the similar mathematical literacy proficiency 
in completing the constructed response test items. The mapping on the students’ mathematical 
literacy proficiency was analyzed based on the type of the test administered and differentiated 
based on the category of each school level (low, moderate and high). Based on the school level, 
the development of the students’ mathematical literacy proficiency for the 8", 9" and 10" grade 
students would be as follows. 
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Figure 3. The development of students’ literacy proficiency based on the grade 
and school level. 
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Figure 3 explains the students’ mathematical literacy proficiency based on the school 
level. The development of the 9" and 10" grade students improve along with the school level; 
the higher the school level, the more improved their development of mathematical literacy 
proficiency. The students’ mathematical literacy proficiency in the high school level is better 
than that in the moderate school level; similarly, the students’ mathematical literacy proficiency 
in the moderate school ability is better than that in the low school level. However, based on the 
figure above, the 8" grade students’ mathematical literacy proficiency in the moderate school 
level is higher than that in the high school level. The mapping on the development of the 
students’ mathematical literacy proficiency based on the grade and gender would be displayed 
in the Figure 4. 

According to the results of PISA International Report (Technical Report), the data from 
the students’ mathematical literacy proficiency were also presented based on the students’ gender. 
Figure 4 showed that the female students’ mathematical literacy proficiency is higher than the 
male students’ mathematical literacy proficiency for the 8", 9" and 10" grade. However, the 
higher the school level was the more equaled the male and the female students’ mathematical 
literacy proficiency in the Province of Yogyakarta Special Region in Indonesia. Moreover, the 
development of the students’ mathematical literacy proficiency based on the type of test and the 
school level altogether in the Figure 5. 
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Figure 4. The development of students’ literacy proficiency based on the grade 
and gender. 


Figure 5 explained the students’ mathematical literacy proficiency based on the school 
level for the multiple-choice test items and the constructed response test items. The 9" and 10" 
grade students’ mathematical literacy proficiency had improved along with the school level 
both for the two types of the test items; the higher the school level was, the more improved 
the 9" and 10% grade students’ mathematical literacy proficiency. The students’ mathematical 
literacy proficiency is higher in the high school level than that in the moderate school level and, 
similarly, the students’ mathematical literacy proficiency is higher in the moderate school level 
than that in the low school level both for the two types of the test items. However, based on the 
figure above, the 8" grade students’ mathematical literacy proficiency in the moderate school 
level is higher than that in the high school level. The reason was that the moderate school level 
that had been sampled was the best moderate school and almost belonged to the high school 
level. In addition, the students’ mathematical literacy proficiency in completing the constructed 
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response test item in the low, moderate and high school level was lower than that in completing 
the multiple-choice test items. 
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Figure 5. The development of students literacy proficiency based on the type of 
test, grade and school level. 
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Figure 6. The level of students’ mathematical literacy proficiency. 


The development of the students’ mathematical literacy proficiency based on the level 
of literacy proficiency is presented in the Figure 6. Based on that figure, most of the 8" grade 
students’ mathematical literacy proficiency is Below Level | in comparison to the 9" and 10% 
grade students. Meanwhile, most of the 9" grade students’ mathematical literacy proficiency 
is Below Level | in comparison to the 10" grade students. These findings showed that the 10" 
grade students’ mathematical literacy proficiency was higher than that of the 8" and 9" grade 
students. However, there were only 10% of the total students who approached the Level 6 
amongst the 8", 9" and 10" grade students. 

The mapping on the development of the students’ mathematical literacy proficiency 
based on the combination of the type of test and the level of literacy proficiency is presented 
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in Figure 7. There were more than 70% students of grade 8 and 9 who scored the literacy 
proficiency in the Level 6 under the multiple-choice test items. On the other hand, there were 
only more than 40% students who scored the literacy proficiency below the Level 1, while there 
were only few students who had scored the literacy proficiency in the Level 4, Level 5 and 
Level 6 under constructed response items. 
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Figure 7. The level of literacy proficiency based on the type of test. 


In addition to observe the pattern of development of students’ literacy proficiency through 
graphics, the next step was to test the development of these abilities using statistical analysis. 
The analysis used for this purpose was analysis of variance (ANOVA), with the dependent 
variable that was literacy proficiency and the independent variables were the students’ grades 
(8, 9, and 10). This analysis tested the null hypothesis that there was similarity means in literacy 
proficiency in grades 8, 9 and 10, with the alternative hypothesis at least one pair was different. 
The results of the analysis with significance level .05 are presented in Table 2. The results of 
this analysis indicate that p-value = .0001 <.05, which indicates the null hypothesis is rejected. 
This represents that there was average students’ literacy proficiency of pairs that were not the 
same between grades. 


Table 2. The analysis of variance result. 














Sum of Squares df Mean Square F value 
Between Groups 140.674 2 70.337 66.731 .0001 
Within Groups 1050.880 997 1.054 
Total 1191.554 999 





To know the different pairs, as well as the sequence of average of students’ literacy 
proficiency, post-hoc analysis was carried out. In this research, post-hoc analysis was carried 
out with Scheffe-test. The full results are presented in Table 3. Based on these results it can be 
obtained that the average literacy proficiency of grade 8 was significantly lower than grade 9 
and grade 10, but students’ literacy proficiency of grades 9 and 10 was not different significantly. 
These results indicate that students’ literacy proficiency from grade 8 to grade 9 increased 
significantly, while from grade 9 to grade 10 it didn’t increase significantly. 
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Table 2. The post-hoc result. 





Multiple Comparisons 





95% Confidence Interval 
(l)Grade (J) Grade Mean Difference (I-J) Std. Error —_ p-value 




















Lower Bound Upper Bound 
9 -.9315173° 0976629  .0001 -1.170931 -.692104 
: 10 -1.0887356" .0953502 0001 -1.322480 -.854991 
8 9315173" .0976629 0001 692104 1.170931 
° 10 -.1572183 0709166  .0860 -.331065 .016629 
8 1.0887356° .0953502 0001 854991 1.322480 
. 9 1572183 0709166  .086 -.016629 .331065 





*. The mean difference is significant at the .05 level. 
Discussion 


The research results showed that in general there had been an improvement on the 
mathematical literacy proficiency from 8" grade to the 9" grade and to the 10" grade. However, 
the development from 9" grade to 10" grade was not significant. In relation to the test item 
design, the literacy proficiency development generated a higher score if it was measured by 
administering the multiple-choice test items than the constructed response test items. Regarding 
the school level, there was a tendency that the school level might influence the achievement 
of mathematical literacy score; the higher the school level was, the higher the mathematical 
literacy score that the students achieved would be. In relation to the gender, within the 8° 
grade, the female students achieved a higher score but in the 9" grade and the 10" grade, the 
achievement of literacy score was almost similar. The 8" grade students were more dominant in 
the “Below Level 1’” “Level 1” and “Level 2” category, while the 9" grade and the 10" grade 
students were more dominant in the “Below Level 1”, “Level 1,” “Level 2,” “Level 3” and 
“Level 4.” 

There was an improvement of literacy from the 8" grade to the 9" grade and to the 
10" grade. The higher the grade was, the higher the students’ achievement was in the score 
of mathematical literacy proficiency. The achievement level of the students’ mathematical 
literacy proficiency was also influenced by the grade. The reason was that the mathematics 
learning materials that the students studied in the 9" grade and the 10" grade are more abundant 
and containing many topics in accordance with the Standards of Content for the Educational 
Implementation issued by the government (Menteri Pendidikan Nasional Indonesia, 2006; 
Menteri Pendidikan dan Kebudayaan Republik Indonesia, 2015). As a result, the higher the 
students’ grade was the more learning materials that supported the improvement of mathematical 
literacy proficiency would be. 

The students’ score of mathematical literacy proficiency would be higher if the score 
was assessed by administering the multiple-choice test items than the constructed response 
test items. The reason was that within the mathematical examination in the school, the teachers 
preferred to administer the multiple-choice test items more than the constructed response test 
items. The multiple-choice test items were administered in the mid-semester examination, the 
final examination, the school final examination, and even in the national examination. 

The school-level indeed influenced the achievement of the students’ mathematical 
literacy proficiency score. The schools that had a high achievement of national examination 
score also had a high score of mathematical literacy proficiency. These results might be caused 
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by the situation that the schools with high achievement usually possessed better input of 
students, better learning process and, as a result, better learning achievement. These findings 
are in line with the opinion of Bohlmann and Pretoria (2008). The tendency that the students’ 
mathematical literacy proficiency in the content of change and relationship, and of uncertainty 
and data was higher than that in the content of quantity, and of shape and space, both based on 
the grade and the school level which might be caused by multiple factors. The reason was that 
the material contents in the 8" grade and the 9" grade dominated by the content of change and 
relation and of uncertainty (Menteri Pendidikan Nasional Indonesia, 2006; Menteri Pendidikan 
dan Kebudayaan Republik Indonesia, 2015). In addition, the content of shape and space 
belonged to the material contents that had been difficult to study than the other contents based 
on the result of national examination (Balitbang Kemendiknas, 2014; 2015; 2016). Regarding 
the influence of gender towards the achievement, in this research, the 8" grade female students 
earned higher score than the 8" grade male students. However, the achievement of male and 
the female students in the 9" grade and the 10" grade was almost similar. These findings are in 
line with the opinion of Ajali and Imoko (2015) but are not quite similar to the opinion of Zhu 
(2007) and Hall (2012). 

Low students’ achievement should be a matter of attention. Several related studies might 
be conducted in order to overcome the low students’ achievement. Since the school quality 
influences the quality of learning achievements (Bohlman & Pretorius, 2008), the improvement 
towards the learning quality and the learning assessment can be pursued through the integration 
of the literacy into the mathematics learning process (Steihilber, 2006) and other subjects. 


Conclusions and Implications 


The results of the research showed that in general there has been the improvement of 
literacy proficiency among the 8th, 9th and 10th grade students. The literacy development would 
generate higher scores if the development had been measured by means of multiple-choice test 
items. There was a tendency that the school level, gender, the form of items, content, context, 
and process literacy would influence the mathematical literacy score. The 8th grade, 9th grade 
and 10th grade students’ literacy proficiency tends to improve although the literacy proficiency 
is still under the mean of mathematical literacy proficiency in the international study. 

In relation to strengthening mathematical literacy proficiency, both the teachers and the 
researchers may develop a test instrument that contains literacy. The learning set contains the 
lesson plan, the student’s worksheet and the assessment sheet. In terms of school assessment, 
the teachers should be benefited by the constructed response test items in that the students will 
be trained to perform critical thinking. By doing so, the quality of literacy learning process 
might be improved and the assessment system that has been applied might drive the students to 
learn about literacy. 

Several strategies can be done to improve literacy skills, so that students’ proficiency 
increase from year to year. The teaching and learning that train literacy can be carried out 
by teachers in schools, starting from elementary school. Of course, the teaching and learning 
literacy needs to be integrated with the curriculum. The various contexts of literacy need to be 
introduced to students, so that students get used to solve many problems. In addition to learning, 
the competence of teachers to train students’ literacy in their schools needs to be considered. 
This is related to the teachers’ role that is the spearhead of the literacy learning. 

The future researches about literacy proficiency can be carried out by other researchers 
related to the results of this research. The students’ difficulties in solving literacy problems 
need to be revealed in detail. Utilizing known difficulties students can follow up and determine 
strategies to overcome them. To identify whether learning that has been carried out by the 
teacher to integrate literacy is also important. The ability of the teacher to carry out learning 
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that trains literacy also needs to be described. In addition, research related to the development 
of students’ literacy proficiency starting from elementary school needs to be conducted, so that 
there is comprehensive progress starting from grade | to grade 12. These studies can then be a 
reference for policies to improve students’ literacy skills at various levels of education. 
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